NITI: Training Integer Neural Networks Using Integer-Only Arithmetic

Low bitwidth integer arithmetic has been widely adopted in hardware implementations of deep neural network inference applications. However, despite the promised energy-efficiency improvements demanding edge applications, use low for training remains limited. Unlike inference, demands high dynamic range and numerical accuracy quality results, making low-bitwidth particularly challenging. To address this challenge, we present a novel framework called NITI that exclusively utilizes arithmetic. stores all parameters accumulates intermediate values as 8-bit integers while using no more than 5 bits gradients. provide necessary during process, per-layer block scaling exponentiation scheme is utilized. By deeply integrating with rounding procedures entropy loss calculation, proposed incurs only minimal overhead terms storage additional computation. Furthermore, hardware-efficient pseudo-stochastic eliminates need external random number generation to facilitate conversion from wider results lower precision storage. Since operates standard storage, it possible accelerate existing operators originally developed commodity accelerators. demonstrate this, an open-source software implementation end-to-end training, native operations modern GPUs presented. In addition, experiments have conducted on FPGA-based accelerator evaluate advantage NITI. When compared equivalent setup implemented floating point arithmetic, degradation MNIST CIFAR10 datasets. On ImageNet, achieves similar state-of-the-art frameworks without relying full-precision floating-point first last layers.